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DEVELOPMENT  RISK  METHODOLOGY  FOR  WHOLE  SYSTEMS  TRADE  ANALYSIS 


1.  INTRODUCTION 

Whole  Systems  Trade  Analysis  (WSTA)  is  a  powerful  decision  tool  used  to  model  the 
relationship  between  design  decisions  and  stakeholder  value  in  order  to  infonn  and  potentially 
influence  requirements  documents  and  associated  specifications.  The  WSTA  tool  (WSTAT)  can 
also  be  used  to  conduct  cost  infonned  trades  analysis  based  on  holistic  design  choices,  while 
understanding  the  opportunity  cost  of  each  choice.  Program  Executive  Office  (PEO)  Ground 
Combat  Systems  (GCS)  developed  the  WSTA  methodology,  and  has  successfully  applied  it  across 
several  programs.  WSTAT  integrates  otherwise  separate  subsystem  models  into  a  holistic  system 
view,  mapping  critical  design  choices  to  consequences  relevant  to  stakeholders,  in  order  to  avoid 
oversimplification,  avoid  sub-optimization,  and  find  the  balance  across  competing  objectives. 
Potential  applications  for  the  tool  include  capability  gap  assessments  (originated  by  the  Joint 
Capabilities  Integration  and  Development  System  (JCIDS))[  1  ],  technology  exploitation, 
requirements  definition,  early  cost  informed  trades,  requirements  analysis,  Analysis  of 
Alternatives  (AoA),  contractor  trades,  and  technology  trade  options.  WSTA  and  its  potential 
applications  supports  and  complies  with  the  Weapon  Systems  Acquisition  Reform  Act  (WSARA) 
of  2009  [2], 

WSTA  has  opened  trade  space  exploration  by  allowing  the  tool  to  evaluate  trillions  of 
potential  system  configurations  to  then  return  a  handful  of  configurations  which  best  meet  design 
and  programmatic  criteria  [3].  WSTA  holistically  displays  system  results  along  with  the  five 
value  dimensions  and  their  consequences.  These  dimensions  are:  performance,  unit  cost, 
operations  &  sustainment  (O&S)  cost,  development  risk,  and  growth  potential.  Identifying  the 
tradeoff  between  these  competing  objectives  is  a  non- trivial  task  and  requires  the  values  and 
preferences  of  key  stakeholders,  including  users  and  Anny  leadership.  Figure  1  depicts  a  notional 
plot  of  the  configurations  which  best  meet  design  and  programmatic  criteria  (these  “best” 
configurations  are  called  Pareto  Optimal  Solutions). 
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Figure  1.  Pareto  Optimal  Solutions. 
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PEO  GCS  requested  U.S.  Army  Materiel  Systems  Analysis  Activity  (AMSAA)  to  conduct 
a  verification  and  validation  (V&V)  of  WSTAT.  In  the  early  stages  of  the  V&V,  it  was 
discovered  that  the  original  risk  rating  and  methodology  did  not  actually  represent  development 
risk,  because  consequences  were  not  considered  (only  likelihood).  As  a  result,  an  appropriate 
development  risk  methodology  needed  to  be  created  for  the  intended  WSTAT  applications.  In 
addition  to  incorporating  consequence  into  the  risk  methodology,  AMSAA  created  a  methodology 
for  the  technology  distributions,  considered  criticality  of  technologies,  developed  a  closed  fonn 
algorithm,  and  created  a  risk  rating  approach.  For  the  WSAT  application,  development  risk  is 
defined  as  the  risk  of  not  delivering  one  or  more  technologies  on  time  at  the  Milestone  B  schedule 
date. 


The  new  development  risk  methodology  contains  a  risk  score,  a  value  score,  and  a  risk 
rating.  The  methodology  was  presented  to  and  accepted  by  PEO  GCS  and  Sandia  National 
Laboratories  (SNL).  The  risk  and  value  scores  are  currently  being  utilized  in  the  WSTA  genetic 
algorithm  to  explore  the  trillions  of  configurations.  Pending  the  availability  of  funding,  the  risk 
rating  algorithm  will  be  incorporated  into  WSTA  and  applied  to  a  final  holistic  mapping  of  the 
“best”  configurations.  The  purpose  of  this  report  is  to  focus  on  how  development  risk  was  created 
for  WSTAT  and  is  intended  for  an  analyst  involved  in  the  development  or  application  of  WSTAT. 


2.  DEFINITION  AND  OVERVIEW  OF  DEVELOPMENT  RISK 

Risk  is  defined  [4]  by  some  undesired  event,  the  likelihood  of  that  event,  and  the 
consequence  if  the  event  occurs.  Based  on  the  definition  of  risk  and  intended  use  of  WSTAT, 
development  risk  contains  these  three  basic  elements: 

1 .  An  event,  which  is  time  or  schedule  related:  Not  delivering  one  or  more  technologies  on 
time  at  the  Milestone  (MS)  B  schedule  date. 

2.  A  likelihood  (L)  for  the  event. 

3.  A  consequence  (C)  if  the  event  occurs  (add  time  to  the  schedule  -  a  schedule  overrun). 

In  order  for  development  risk  to  be  incorporated  into  the  existing  WSTAT  framework, 
AMSAA  had  to  develop  an  approach  for  detennining  a  risk  score  that  considered  the  three 
elements  listed  above.  Prior  to  developing  a  method  for  determining  a  risk  score,  AMSAA 
improved  the  existing  technology  time  distributions  utilized  in  WSTAT.  In  addition,  AMSAA 
developed  a  method  for  measuring  technology  criticality  using  consequence.  The  technology  time 
distributions  and  technology  criticality  were  then  incorporated  in  a  Monte  Carlo  simulation  to 
detennine  a  risk  score.  Since  the  Monte  Carlo  simulation  would  significantly  increase  the  runtime 
for  WSTAT,  a  closed  form  algorithm  was  developed  to  instead  implement  in  WSTAT.  Lastly,  a 
method  for  creating  a  risk  rating  (high,  moderate,  low)  was  developed.  The  following  sections  of 
this  report  describe  the  improvements  for  the  technology  time  distributions,  incorporation  of 
criticality  as  a  component  of  consequence,  creation  of  a  risk  score,  development  of  a  closed  form 
algorithm,  and  creation  of  a  risk  rating. 
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3.  CREATION  OF  AN  IMPROVED  TIME  ASSESSMENT 


3.1  Existing  Time  Assessment  Approach. 

WSTAT  produces  trillions  of  potential  system  configurations,  each  of  which  are  composed 
of  many  underlying  technologies.  Development  times  for  these  technologies  (specifically,  the 
time  to  reach  MS  B)  within  each  system  configuration  are  what  drives  the  development  risk  score 
and  rating.  The  initial  steps  of  applying  the  WSTA  development  risk  process  are  to  identify  all 
possible  technologies  of  interest  for  the  system  under  consideration,  and  to  identify  subject  matter 
experts  (SMEs)  that  are  knowledgeable  regarding  the  current  status,  potential  risks,  and  estimated 
development  times  associated  with  each  technology.  The  established  WSTAT  framework  utilizes 
elicitation  techniques  with  these  SMEs  to  evaluate  the  10th,  50th,  and  90th  percentile  development 
times  to  reach  TRL  6  for  each  technology.  Figure  2  illustrates  a  typical  time  assessment  for  one 
notional  technology.  This  time  assessment  is  a  probability  distribution  [7]  where  the  area  under 
the  curve  totals  1.0. 


Technology  1 


Time  (in  months) 


0.1 


Percentile  0 


12  30  40  45 


10  50  90  100 


Figure  2.  Initial  Completion  Time  Assessment. 


The  percentile  time  values  (in  months)  of  12,  30,  and  40  months  (or  calendar  dates  could 
be  used)  are  assessed  from  the  SMEs  at  a  workshop.  Using  these  percentile  time  values,  the 
existing  WSTAT  framework  determines  that  the  100th  percentile  is  45  months.  As  shown  in  the 
probability  distribution  in  Figure  2,  the  area  under  the  curve  totals  1.0. 


3.2  Improved  Time  Assessment  Approach. 

Notice  that  the  probability  distribution  in  Figure  2  has  no  “right  tail”.  It  is  documented  that 
time  distributions  [5]  usually  have  a  right  tail,  since  uniformity  in  the  tail  of  a  time  assessed 
distribution  is  not  realistic.  Figure  3  depicts  the  improved  notional  distribution  previously  shown 
in  Figure  2. 
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Figure  3.  Improved  Completion  Time  Assessment. 


The  height  of  the  last  rectangle  for  the  original  probability  distribution  (Figure  2)  is  cut  in 
half  horizontally  and  the  length  of  the  rectangle  (Figure  3)  becomes  twice  as  long.  The  new  100th 
percentile  now  becomes  50  months.  This  creates  a  tail  in  Figure  3  which  better  represents  the 
completion  time  distribution.  The  area  of  this  last  rectangle  remains  0. 1  by  design. 


4.  MEASURING  CRITICALITY  USING  CONSEQUENCE 

Criticality  of  a  developmental  technology  can  be  incorporated  into  the  risk  of  that 
technology.  The  concept  of  criticality  is  illustrated  in  this  section  by  adjusting  the  consequence  or 
schedule  overrun  based  on  the  criticality  level. 

Suppose  two  technologies  both  overrun  the  schedule  by  the  same  number  of  months  (e.g., 
eight  months)  and  one  technology  is  critical  and  the  other  is  not.  While  both  technologies  will 
probably  take  eight  months  to  complete,  the  critical  technology  carries  much  more  schedule  risk 
than  the  lower  critical  technology  because  the  more  critical  technologies  could  impact  key 
program  activities,  such  as: 

•  Cause  a  MS  B  or  C  date  to  change  if  an  overrun  occurred, 

•  Cause  more  integration,  manufacturing,  and  development  issues  in  the  future,  or 

•  Cause  operational  testing  &  evaluation  issues  due  to  being  a  novel  technology. 

The  three  impacts  listed  above  should  be  considered  when  determining  whether  or  not  a 
technology  is  critical.  A  non-critical  technology  is  not  involved  in  any  of  these  three  events, 
whereas  a  critical  technology  plays  a  role  in  at  least  one  of  these  events.  In  the  previous  example 
with  two  technologies,  the  critical  technology  will  have  a  higher  consequence  than  the  non-critical 
technology.  Consequence  in  this  case  is  measured  by  schedule  overrun,  or  how  much  the  schedule 
slipped  beyond  the  MS  B  date  due  to  that  technology.  Normally,  critical  and  non-critical 
technologies  would  not  have  the  same  consequence.  Flence,  the  non-critical  technology  should 
have  a  consequence  <  8.  The  process  or  mechanism  to  determine  this  involves  reducing  the 
consequence  (slippage  or  overrun  of  “8”)  for  the  less  critical  technologies.  The  reduction  amount 
is  called  “slack”.  The  following  are  two  possible  criticality  flags: 


4 


•  High  (i.e.,  the  technology  needs  all  of  the  overrun  time  -  no  slack)  or 

•  Low  (i.e.,  the  technology  does  not  need  all  of  the  overrun  time  -  slack  is  required) 

Slack  would  be  applied  to  the  low  criticality  flags  for  a  given  technology.  This  means  that 
if  a  technology  has  a  low  criticality  flag,  then  some  amount  of  “slack”  is  subtracted  from  the 
schedule  overrun  (or  consequence)  for  that  technology.  Slack  is  based  on  number  of  months  and 
is  determined  by  the  user  based  on  its  product  structure  and  technology  type  (i.e.  a  user  created 
table).  For  example,  suppose  a  certain  system  configuration  has  one  developmental  technology 
(DT)  with  a  completion  time  distribution.  DT  #1  is  assigned  a  low  criticality  with  a  6  month  slack 
constant  (based  on  its  product  structure  and  type).  Suppose  DT  #1  took  18  months  to  complete 
and  MS  B  was  in  10  months.  Figure  4  depicts  this  notional  example. 


Schedule  Overrun  (SO)  =  8  or  Consequence;  if  High  Criticality 

_ Jl _  ~ 

18-  10  =  8 

MS  B  = 

18 

y  | 

Adjusted  SO  for  a  6  month 

1 0  slack  factor  =8-6  =  2 

Figure  4.  Criticality  and  Schedule  Overrun. 


Since  DT  #1  was  assigned  a  low  criticality,  then  the  schedule  overrun  (SO)  was  reduced 
from  8  months  to  2  months.  If  DT  #1  had  been  assigned  a  high  criticality,  then  the  SO  would  have 
been  the  full  8  months. 


5.  TRANSFORMATION  OF  TIME  ASSESSED  DISTRIBUTIONS 

This  section  addresses  how  the  criticality  assignment  affects  the  time  assessed  distribution 
for  a  given  DT.  To  illustrate  this  concept,  a  notional  example  will  be  used.  Suppose  a  DT  (e.g., 
radio)  was  originally  assessed  to  reach  TRL  6  with  the  uniform  distribution  depicted  below  in 
Figure  5  and  MS  B  =  30  months. 


Radio 


0 


MS  B  =  30 


100  months 


Figure  5.  Radio  TRL  6  Time  Assessment. 


Since  the  maximum  time  is  100  months,  the  height  of  the  rectangle  in  Figure  5  must  be  1/100  = 
0.01  because  the  area  of  this  rectangle  must  be  1.0.  If  the  radio  was  assigned  a  low  criticality, 
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with  slack  constant  of  K  =  5  months  determined  by  the  user  based  on  the  radio  family  of 
technologies  (this  is  a  different  notional  example  than  in  section  4).  Then  the  original  time 
assessed  distribution  in  Figure  5  transforms  into  the  following  distribution,  depicted  below  in 
Figure  6  (based  on  the  criticality  rules  from  Section  4). 


Figure  6.  Transformed  Radio  TRL  6  Time  Assessment. 


Note  that  the  area  of  the  two  rectangles  are  0.3  and  0.65  (the  height  of  each  rectangle 
remains  at  0.01)  and  the  probability  of  completing  in  exactly  30  months  is  0.05  because  of  the  5 
month  slack  shift.  Note:  Times  >  30  become:  30  +  schedule  overrun  after  slack  of  5  months  is 
applied.  Therefore,  5%  of  the  original  times  in  Figure  5  (times  between  30  -  35)  will  be  30,  since 
the  schedule  overruns  will  be  zero  after  slack  is  applied  (negative  schedule  overruns  must  be  0  or 
bigger  -  time  is  a  positive  continuous  distribution). 


6.  CREATION  OF  A  RISK  SCORE 

The  development  risk  methodology  consists  of  a  risk  score,  a  value  score,  and  a  risk  rating. 
This  section  illustrates  how  the  risk  score  is  developed  by  extending  the  example  from  Figure  4  in 
Section  4.  Suppose  a  certain  system  configuration  has  two  DT’s,  each  with  different  completion 
time  distributions.  DT  #1  is  assigned  a  low  criticality  (with  a  6  month  slack  constant)  and  DT  #2 
is  high  criticality  (no  slack).  Suppose  DT  #1  took  18  months  to  complete  and  DT  #2  took  1 1 
months  to  complete,  where  MS  B  was  in  10  months.  Let  this  be  one  instance  or  run  of  a  Monte 
Carlo  simulation  [6].  The  maximum  schedule  overrun  for  these  two  DT’s  is  MAX(2,  1)  =  2, 
illustrated  in  Figure  7  example. 


PTtfl 

_ 1 _  Schedule  Overrun 

18-  10  =  8 

18  * 

SO  =  8-6  =  2  1 

MS  B 

ii  - 

MAX  =  2 

=  10 

Figure  7.  Notional  System  Configuration  with  Two  DT’s. 


The  next  step  is  to  perform  this  Monte  Carlo  simulation  a  large  number  of  times  (e.g.,  1000 
times)  to  create  a  distribution  of  schedule  overruns,  where  no  overrun  is  assigned  a  0.  This 
resulting  distribution  is  called  the  risk  distribution.  Suppose  the  95th  percentile  of  this  distribution 
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is  P.95  =  3.75  months.  A  notional  risk  distribution  and  P.95  look  like  the  following  graph 
depicted  in  Figure  8. _ 


.62 

P.95 

.33 

=  3.75 

.05 

1  =  .62  +.33  +  .05 

1 

0 

3.75 

10 

Figure  8.  Notional  Risk  Distribution. 


The  sum  of  the  probabilities  for  all  schedule  overruns  (0  overrun  implies  system  completed 
before  MS  B)  sums  to  1.0,  therefore  the  risk  distribution  is  a  probability  distribution  [5].  This  risk 
distribution  incorporates  all  three  elements  of  risk  that  were  discussed  in  Section  2  (an  event,  the 
likelihood  of  the  event,  and  the  consequence  of  the  event).  The  P.95  value  represents  the  risk  level 
for  a  particular  system  configuration.  Higher  values  of  P.95  are  riskier  than  lower  values.  The 
idea  to  focus  on  is  the  relative  risk  of  all  system  configurations  in  relationship  to  each  other.  The 
95th  percentile  was  chosen  to  assure  this  relative  risk  spread  -  any  large  percentile  (i.e.,  85th,  90th, 
etc.)  may  have  also  worked  because  the  relative  rankings  of  risk  will  not  change.  P.95  now 
becomes  the  WSTA  development  risk  score  and  represents  the  risk  of  a  particular  system 
configuration. 


7.  CLOSED  FORM  SOLUTION  TO  DEVELOPMENT  RISK 

Since  a  WSTA  development  risk  score  is  needed  for  each  of  the  trillions  of  system 
configurations,  it  is  not  practical  for  the  tool  to  run  that  many  Monte  Carlo  simulations.  This 
section  documents  a  closed  form  solution  that  was  developed  to  address  this  problem.  To 
illustrate  this  algorithm  the  risk  distribution  from  Figure  8  will  be  used.  This  risk  distribution 
represents  a  single  system  configuration.  Suppose  the  probability  of  no  schedule  overrun  (0)  for 
this  configuration  is  0.62,  and  MS  B  =  10  months.  This  0.62  probability  is  simply  the  product  of 
the  probabilities  that  each  DT  achieves  TRL  6  in  less  than  10  months.  This  configuration  has  two 
DT’s,  where  the  probability  of  no  schedule  overrun  (0)  for  DT  #1  and  DT  #  2  is  0.8  and  0.775 
respectively,  for  MS  B  =  10  months,  thus  0.8  *  0.775  =  0.62. 

The  goal  is  to  find  the  95th  percentile  (P.95)  of  this  particular  risk  distribution.  If  the 
probability  of  no  schedule  overrun  (0)  for  this  configuration  was  0.95,  and  MS  B  =  10  months, 
then  P.95  =  0.  To  find  this  P.95  without  using  simulation,  it  is  necessary  to  utilize  a  binary  search 
algorithm  to  find  the  number  of  months  (X)  when  the  probability  of  no  schedule  overrun  (0)  for 
this  configuration  is  0.95.  After  conducting  the  binary  search,  suppose  X  =  13.75  months. 
Therefore  the  P.95  of  the  risk  distribution  in  Figure  8  is  13.75  -  10  =  3.75  months. 
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8.  RISK  VALUE  SCORE 


The  existing  WSTAT  framework  utilizes  value  scores  and  preferences,  which  need  to  be 
evaluated  for  the  trillions  of  possible  risk  scores.  To  do  this  the  user  decides  what  the  value  curve 
will  be  based  on  their  preferences.  For  example,  a  low  preference  to  P.95  scores  between  5  and  10 
months,  medium  preference  for  scores  between  3  and  5  months,  and  high  preference  or  value  for 
scores  between  0  and  3  months.  A  plot  of  this  value  curve  might  look  like  the  following  graph 
shown  in  Figure  9.  The  risk  and  value  scores  are  ordered  worst  to  best  on  their  respective  axes. 
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5  3  0 

Risk  Score  =  R95 


Figure  9.  Value  Curve. 


Note  that  if  a  different  percentile  was  used,  one  may  still  conclude  the  same  value  curve. 
The  value  score  is  then  used  in  the  WSTA  genetic  algorithm  to  explore  the  trillions  of 
configurations  to  select  the  “best”  configurations. 

9.  CREATION  OF  A  RISK  RATING 

Once  the  best  system  configurations  are  selected  from  the  WSTA  genetic  algorithm,  a  risk 
rating  needs  to  be  assigned  to  each  of  the  remaining  configurations.  This  section  discusses  the 
methodology  behind  the  steps  to  create  a  risk  rating  for  an  alternative. 

Suppose  P.95  =  6.8  months  and  L  =  0.9  (1  -  L  =  0.1).  L  is  the  likelihood  of  not  delivering 
one  or  more  technologies  on  time  at  the  MS  B  schedule  date.  Figure  10  is  an  approximation  for 
the  original  risk  distribution.  The  P.  100  (100th  percentile)  of  8. 1  months  was  detennined  by  the 
maximum  of  the  100th  percentile  for  all  DT  time  distributions. 
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Figure  10.  Original  Risk  Distribution. 
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The  original  risk  distribution  in  Figure  10  utilizes  both  L  (1  -  .1  =  .9)  and  consequence  (C) 
within  the  same  distribution.  Consequence  is  measured  as  schedule  overruns.  Let’s  separate  L 
and  C  into  two  parts  by  only  considering  the  distribution  of  schedule  overruns.  Figure  1 1  shows 
how  the  original  risk  distribution  from  Figure  10  is  converted  into  a  consequence  distribution. 

The  median  of  the  consequence  distribution  is  3.6  months.  Assign  C  =  3.6  and  L  =  0.9. 


The  final  step  is  to  assign  a  risk  rating  (high  =  red;  moderate  =  yellow;  or  low  =  green) 
based  on  the  L  and  C  values.  The  preferences  of  the  WSTA  user  ideally  need  to  be  reflected  in  the 
risk  matrix  and  L  and  C  value  bins.  Figure  12  shows  the  DoD  Risk  Reporting  Matrix  as  a  starting 
point. 
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Consequence 

Figure  12.  DoD  Risk  Reporting  Matrix. 


The  preferences  of  the  user  are  reflected  by  the  location  of  the  three  colors  in  Figure  12 
and  the  definitions  of  the  L  and  C  bins.  Figure  12  is  an  example  of  the  DoD  Risk  matrix  and  is  not 
necessarily  the  preferences  of  the  WSTA  users.  Next,  it  is  necessary  to  define  bins  for  each  of  the 
five  L  and  C  levels.  An  example  of  a  risk  matrix  with  the  L  and  C  bins  defined  is  shown  in  Figure 
13. 
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Figure  13.  Risk  Matrix  with  Bin  Definitions. 


L  and  C  definitions  to  aid  in  defining  the  bins  may  look  like  the  following  two  tables 
(Tables  1  and  2).  The  Likelihood  and  Consequence  level  definitions  may  be  tailored  for  specific 
WSTA  applications. 


Table  1.  An  Example  of  Likelihood  Definitions. 


Level 

Likelihood 

Probability  Ranges 

1 

Not  Likely 

L  <=  20% 

2 

Low  Likelihood 

20%  <  L  <=  40% 

3 

Likely 

40%  <  L  <=  60% 

4 

Highly  Likely 

60%  <  L  <=  80% 

5 

Near  Certainty 

L  >  80% 

Notice  in  this  example  (Table  1)  that  the  likelihood  values  are  evenly  divided  -  this  may 
not  always  be  the  preference. 
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Table  2.  An  Example  of  Consequence  Definitions. 


Level 

Schedule 

1 

Negligible  schedule  slip. 

2 

Schedule  slip,  but  able  to  meet  key  dates  and  has  no  significant 
impact  to  slack  on  critical  path. 

3 

Schedule  slip  that  impacts  ability  to  meet  key  dates  and/or 
significantly  decreases  slack  on  critical  path. 

4 

Will  require  change  to  program  or  project  critical  path. 

5 

Cannot  meet  key  program  or  project  milestones. 

If  the  alternative  with  L  =  0.9  and  C  =  3.6  months  is  applied  to  the  risk  matrix  in  Figure  13, 
the  risk  rating  would  be  moderate  (yellow). 

10.  COMPARISON  OF  RISK  METHODOLOGIES  USED  FOR  WSTA  AND  AoA’s 

During  the  creation  of  the  WSTA  development  risk  methodology,  many  risk  options  were 
explored.  The  creation  of  the  WSTA  risk  methodology  led  to  significant  improvements  and 
additions  to  AMSAA’s  independent  schedule  risk  assessments  that  support  AoA’s.  Some  of  these 
enhancements  were  related  to  schedule  risk  ratings,  integrated  modeling,  tradespace  analysis,  and 
joint  probability  statements. 

The  risk  methodologies  for  WSTAT  and  AMSAA’s  risk  assessments  for  AoA’s  have 
different  intended  uses,  although  there  are  many  similarities.  The  WSTAT  risk  methodology  is 
appropriate  for  pre-selecting  the  best  system  configurations/altemative  designs,  but  is  not  intended 
to  replace  AMSAA’s  AoA  Risk  Methodology.  Figure  14  shows  a  comparison  of  risk 
methodologies: 
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WSTAT  Development  Risk  Methodology 

>  Intended  Use:  Pre-AoA  to  assess  trillions  of 
potential  system  configurations. 

1.  Completion  time  assessed  using  TRL  (MS  B 
target  date)  for  all  technologies  by  eliciting  10th, 
50th  and  90th  percentiles  from  SMEs. 

2.  Criticality  of  technologies  is  established  based  on 
impact  to  schedule  (High  or  Low). 

3.  Low  criticality  is  addressed  with  slack  constant 
(original  system  configuration  level  performance 
is  still  100%  as  planned). 

4.  System  configuration  level  risk  assessment 
(consequence  is  schedule  overrun)  is  done  at  MS 
B  to  create  Risk  /  Value  Score  &  Risk  Rating. 


Figure  14.  WSTAT  Development  & 


AoA  Schedule  Risk  Methodology 

>  Intended  Use:  AoA  to  assess  a  limited  number  of 
potential  alternatives  (~4-6). 

1 .  Completion  time  assessed  using  TRL,  IRL,  & 
MRL  (MS  B  &  C  target  dates)  by  eliciting 

triangular  distributions  (min,  max,  most  likely) 
for  key  technologies  from  SMEs  during  a  Risk 
Workshop. 

2.  Critical  technologies  are  established  based  on 
assessment  of  whether  a  technology  is  ‘key’  which 
means  potentially  having  an  impact  to  the 
schedule. 

3.  Non-critieal  technologies  are  not  considered. 

4.  Alternative  level  risk  assessment  (consequence  is 
schedule  overrun)  is  done  at  MS  B  &  C  to  assign 

Risk  Rating. 

AoA  Schedule  Risk  Methodologies. 


11.  CONCLUSION 

AMSAA  was  tasked  with  conducting  a  V&V  of  WSTAT.  In  the  early  stages  of  the  V&V 
for  development  risk,  it  was  discovered  that  the  original  risk  rating  and  methodology  did  not 
actually  represent  development  risk,  because  consequences  were  not  considered  (only  likelihood). 
As  a  result,  an  appropriate  development  risk  methodology  needed  to  be  created  for  the  intended 
WSTAT  applications.  In  addition  to  incorporating  consequence  into  the  risk  methodology, 
AMSAA  created  a  right  tail  methodology  for  the  technology  distributions,  considered  criticality  of 
technologies,  developed  a  closed  form  algorithm,  and  created  a  risk  rating  approach. 

The  development  risk  methodology  was  designed  to  address  the  requirement  to  assess  risk 
within  the  WSTA  framework.  This  requirement  was  a  risk  score,  value  score,  and  a  risk  rating 
level  for  each  of  the  many  system  configuration  options.  The  risk  and  value  scores  are  currently 
being  utilized;  however,  the  risk  rating  algorithm  requires  additional  work  to  implement  and  is  not 
currently  utilized.  Implementation  of  the  risk  rating  algorithm  will  be  conducted  as  part  of  future 
specific  customer  applications  of  WSTAT.  The  information  learned  from  creating  the 
development  risk  methodology  also  led  to  many  enhancements  to  AMSAA’ s  independent 
schedule  risk  assessments  that  support  AoA’s. 
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